Image processing apparatus, image processing method, and storage medium

ABSTRACT

An image processing apparatus includes a setting unit configured to set a virtual light source for a captured image; a brightness correction unit configured to correct brightness of a partial region of an object using the virtual light source set by the setting unit; an attribute detection unit configured to detect an attribute of the partial region; a glossy component generation unit configured to generate a glossy component that is to be applied to the partial region, according to the attribute of the partial region detected by the attribute detection unit; and a glossy appearance correction unit configured to correct a glossy appearance of the partial region using the glossy component generated by the glossy component generation unit.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of application Ser. No. 15/146,265,filed May 4, 2016 the entire disclosure of which is hereby incorporatedby reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing apparatus,particularly to an image processing apparatus that corrects the glossyappearance of an input image.

Description of the Related Art

Conventionally, a technique (relighting) of irradiating an object in animage with light from a virtual light source and thereby correcting thebrightness of the object is known (Japanese Patent Laid-Open No.2010-135996). This technique can brighten a dark region such as a shadowmade by ambient light, and can correct an image so that an objectpresent in, for example, a blocked-up shadow can be recognized.

For example, Japanese Patent Laid-Open No. 2010-135996 discloses alighting technique in which a captured image is subjected topseudo-lighting processing. Specifically, a region that has a luminancelower than the average luminance of an entire face region is extractedas a shadow region. Then, the luminosity of the extracted shadow regionis increased. Accordingly, it is possible to suppress the shadow of theobject.

However, although the technique described in Japanese Patent Laid-OpenNo. 2010-135996 can adjust the image quality of a region thatcorresponds to shade or shadow with relighting by irradiating the regionwith light from a virtual light source, it is not possible to adjust theglossy appearance caused by the relighting.

SUMMARY OF THE INVENTION

The present invention was made in view of the above-described problem,and enables, when correcting the brightness of an object in an image byirradiating it with light from a virtual light source, adjustment of theglossy appearance according to the object.

According to a first aspect of the present invention, there is providedan image processing apparatus comprising: a setting unit configured toset a virtual light source for a captured image; a brightness correctionunit configured to correct brightness of a partial region of an objectusing the virtual light source set by the setting unit; an attributedetection unit configured to detect an attribute of the partial region;a glossy component generation unit configured to generate a glossycomponent that is to be applied to the partial region, according to theattribute of the partial region detected by the attribute detectionunit; and a glossy appearance correction unit configured to correct aglossy appearance of the partial region using the glossy componentgenerated by the glossy component generation unit.

According to a second aspect of the present invention, there is providedan image processing method comprising: setting a virtual light sourcefor a captured image; correcting brightness of a partial region of anobject using the virtual light source set in the setting; detecting anattribute of the partial region; generating a glossy component that isto be applied to the partial region, according to the attribute of thepartial region detected in the attribute detecting; and correcting aglossy appearance of the partial region using the glossy componentgenerated in the glossy component generating.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a configuration of a digitalcamera according to a first embodiment of the present invention.

FIG. 2 is a block diagram illustrating a configuration of an imageprocessing unit of the first embodiment.

FIG. 3 is a diagram schematically illustrating the relationship betweenan object and a virtual light source.

FIG. 4 is a flowchart illustrating relighting parameter settingprocessing.

FIG. 5 is a block diagram illustrating a configuration of a relightingprocessing unit of the first embodiment.

FIG. 6 is a flowchart illustrating an operation of glossy appearancecorrection processing of the first embodiment.

FIG. 7 is a diagram illustrating an example of correspondence betweenface/organ detection results and object attributes.

FIG. 8 is a block diagram illustrating a configuration of a relightingprocessing unit according to a second embodiment.

FIG. 9 is a flowchart illustrating an operation of glossy appearancecorrection processing of the second embodiment.

FIG. 10 is a block diagram illustrating a configuration of a relightingprocessing unit according to a third embodiment.

FIG. 11 is a flowchart illustrating an operation of glossy appearancecorrection processing of the third embodiment.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, exemplary embodiments of the present invention will bedescribed in detail with reference to the accompanying drawing. Notethat the embodiments below will describe an example in which an imageprocessing apparatus according to the present invention is applied to adigital camera. Note that “digital camera” refers to an electronicdevice that has a function of capturing an image using a photoelectricconversion element, and encompasses a suitable electronic device such asa mobile phone, a game console, or a personal computer that has or usesa camera. Furthermore, the present invention does not necessarily havethe image capturing function, and the image processing apparatusaccording to the present invention is applicable to any electronicdevice that is capable of image processing.

First Embodiment

FIG. 1 is a block diagram illustrating an example of a configuration ofa digital camera 100 according to a first embodiment of the presentinvention.

In FIG. 1, a lens group 101 is a zoom lens including a focus lens. Ashutter 102 having the diaphragm function is provided between the lensgroup 101 and an image capturing unit 103. The image capturing unit 103includes an image sensor represented by a CCD/CMOS image sensor, whichconverts an optical image formed on an image capturing surface by thelens group 101 into electrical signals for respective pixels. An A/Dconverter 104 converts the analog signals output by the image capturingunit 103 into digital signals (image data).

An image processing unit 105 subjects the image data output from the A/Dconverter 104 to various types of image processing such as colorinterpolation (demosaicing), white balance adjustment, γ correction,contour enhancement, noise reduction, and color correction. An imagememory 106 temporarily stores the image data. A memory control unit 107controls reading and writing from and to the image memory 106. A D/Aconverter 108 converts the image data into analog signals. A displayunit 109 includes a display device such as a LCD or an organic ELdisplay, and displays various types of GUIs, a live view image, an imagethat is read from a recording medium 112 and is reproduced, and thelike. A codec unit 110 encodes the image data stored in the image memory106 by a predetermined method so as to record that data in the recordingmedium, or decodes encoded image data included in an image file for thepurpose of, for example, display.

An interface (I/F) 111 mechanically and electrically connects thedetachable recording medium 112 such as, for example, a semiconductormemory card or a card-type hard disk to the digital camera 100. A systemcontrol unit 50 may be, for example, a programmable processor such as aCPU or MPU. The system control unit 50 executes programs stored in, forexample, a nonvolatile memory 121 or a built-in nonvolatile memory andcontrols required blocks and circuits, thereby realizing the functionsof the digital camera 100. A face/organ detection unit 113 detects, froma captured image, a region in which a face is captured (face detection),and the positions of organs of the face such as the eyes, nose, mouth,and cheeks. A relighting processing unit 114 irradiates an object in theimage with light from a virtual light source, and thereby performsprocessing for correcting the brightness of the object (relightingprocessing).

An operation unit 120 is a collection of buttons and switches that areused by a user to input various instructions to the digital camera 100.

The nonvolatile memory 121 may be, for example, an EEPROM, which iselectrically erasable and storable, or the like. The nonvolatile memory121 stores not only various types of setting values and GUI data, butalso programs to be executed by the system control unit 50 if the systemcontrol unit 50 is an MPU or CPU.

A system memory 122 is used for expansion of constants and variables foroperation of the system control unit 50, a program read from thenonvolatile memory 121, and the like.

The following will describe the operation of digital camera 100 at thetime of image capturing.

For example, the image capturing unit 103 photoelectrically converts,using the image sensor, an object image that is formed on the imagecapturing surface by the lens group 101 while the shutter 102 is open,and outputs the photoelectrically converted object image as analog imagesignals to the A/D converter 104. The A/D converter 104 converts theanalog image signals output from the image capturing unit 103 intodigital image signals (image data), and outputs the converted digitalimage signals to the image processing unit 105.

The image processing unit 105 subjects the image data from the A/Dconverter 104 or image data from the memory control unit 107 to varioustypes of image processing such as color interpolation (demosaicing), γcorrection, contour enhancement, noise reduction, and color correction.

Furthermore, the image processing unit 105 performs predeterminedevaluation value calculation processing based on a result of detectionof a position or region of the face/organ that was obtained by theface/organ detection unit 113 or the captured image data, and the systemcontrol unit 50 performs exposure control and distance measurementcontrol based on the obtained evaluation value result. The imageprocessing unit 105 further performs automatic white balance (AWB)adjustment using the captured image data. Accordingly, the digitalcamera 100 of the present embodiment performs TTL (through-the-lens)type AF (automatic focusing) processing, AE (automatic exposure)processing, and AWB (automatic white balance) processing. Furthermore,the system control unit 50 controls relighting by the relightingprocessing unit 114 and processing for correcting the glossy appearancein the face, with reference to a result of detection of a position orregion of the face/organ that was obtained by the face/organ detectionunit 113.

The image data output from the image processing unit 105 is written inthe image memory 106 via the memory control unit 107. The image memory106 stores the image data output from the image capturing unit 103 orimage data to be displayed on the display unit 109.

Furthermore, the D/A converter 108 converts data for image display thatis stored in the image memory 106 into analog signals, and supplies theconverted analog signals to the display unit 109. The display unit 109performs display corresponding to the analog signals from the D/Aconverter 108 on the display device such as a LCD.

The codec unit 110 encodes the image data recorded in the image memory106 based on a standard such as JPEG or MPEG. The system control unit 50adds a predetermined header or the like to the encoded image data toform an image file, and records the image file in the recording medium112 via the interface 111.

Note that existing digital cameras ordinarily capture a moving image andcontinue to display the captured moving image on the display unit 109 inan image capturing standby state, and thus the display unit 109functions as an electronic viewfinder (EVF). In this case, the shutter102 is set to be in an open state, and an image is captured at, forexample, 30 frames/second using a so-called electronic shutter of theimage capturing unit 103.

Also, if a shutter button included in the operation unit 120 is pressedhalfway down, the above-described AF or AE control will be performed,and if the shutter button is pressed all the way down, actual imagecapturing will be performed to capture a still image for storage and thecaptured still image will be recorded in the recording medium 112.Furthermore, if moving image capturing is instructed using a movingimage capturing button, recording of a moving image in the recordingmedium 112 is started.

In addition to the above-described basic operations, the system controlunit 50 executes processing of the present embodiment that will bedescribed later by executing the above-described programs stored in thenonvolatile memory 121. In this context, “programs” refer to programsfor executing various flowcharts of the present embodiment that will bedescribed later. In this case, constants and variables for the operationof the system control unit 50, programs read from the nonvolatile memory121, and the like are expanded on the system memory 122.

The following will describe the details of the image processing unit 105with reference to FIG. 2. FIG. 2 is a block diagram illustrating aconfiguration of the image processing unit 105.

In FIG. 2, the image processing unit 105 includes a luminance/colorsignal generation unit 200, a contour enhancement processing unit 201, aluminance gamma processing unit 202, a WB (white balance) amplifyingunit 203, a color conversion processing unit 204, a color gammaprocessing unit 205, and a color-difference signal generation unit 206.

The following will describe processing of the image processing unit 105.Image signals from the A/D converter 104 of FIG. 1 are input to theimage processing unit 105. The image signals input to the imageprocessing unit 105 are input to the luminance/color signal generationunit 200. The luminance/color signal generation unit 200 subjects theinput RGB image data in a Bayer arrangement to synchronizationprocessing so as to generate color signals R, G, and B. Furthermore, theluminance/color signal generation unit 200 generates a luminance signalY based on the RGB signals. The luminance/color signal generation unit200 outputs the generated luminance signal Y to the contour enhancementprocessing unit 201, and outputs the color signals R, G, and B to the WBamplifying unit 203.

The contour enhancement processing unit 201 performs contour enhancementprocessing on the luminance signal, and outputs the processed signal tothe luminance gamma processing unit 202. The luminance gamma processingunit 202 performs gamma correction on the luminance signal Y, andoutputs the luminance signal Y to the image memory 106.

The WB amplifying unit 203 applies a gain to the RGB color signals basedon a white balance gain value that is calculated by the system controlunit 50 using processing that will be described later, and adjusts thewhite balance. The color conversion processing unit 204 performs matrixcalculation or the like on the RGB signals, and converts the RGB signalsso that they have a desired color balance. The color gamma processingunit 205 performs gamma correction on the RGB color signals. Thecolor-difference signal generation unit 206 generates an R−Y signal anda B−Y signal, which are color-difference signals, based on the RGBsignals. The Y signal, the R−Y signal, the B−Y signal that are imagesignals output to the image memory 106 are compressed and encoded by thecodec unit 110, and are recorded in the recording medium 112.

The following will describe the preparation operation before theoperation of the relighting processing unit 114. In the presentembodiment, it is assumed that as an example of the relightingprocessing, a captured image of a human face as shown in FIG. 3 isirradiated with light from a virtual light source that is set inaccordance with an instruction of a user, and the brightness andshade/shadow, as well as the glossy appearance in object regions such asthe skin, hair, and pupils are corrected.

Prior to the operation of the relighting processing unit 114, the systemcontrol unit 50 calculates control parameters that are to be set for therelighting processing unit 114, and sets the calculated controlparameters for the relighting processing unit 114. The operation forsetting the parameters at the time of the relighting processing will bedescribed with reference to the flowchart of FIG. 4.

In step S501 of FIG. 4, an operation of a user on the operation unit 120is accepted. Specifically, with an operation of the user on theoperation unit 120, the relighting processing is selected from a menu(not shown), and parameters for the relighting processing are input. Inthe present embodiment, it is assumed that the position of the virtuallight source, the intensity (α) of the virtual light source, and thediffusion property (β) of the virtual light source are input by the useras the parameters for the relighting processing. Note that a method bywhich the user inputs the parameters may be a method by which the userselects parameter values from preset values.

In step S502, the intensity of the light source and the diffusionproperty of the light source that were input by the user are set for thecontrol parameters of the relighting processing unit.

An example of setting the position of the virtual light source and acentral irradiation position are shown in FIG. 3. In FIG. 3, thereference numeral 701 denotes the position of the virtual light source,and the reference numeral 702 denotes the central irradiation positionof light from the virtual light source. Furthermore, the referencenumeral 703 denotes the irradiation range of light from the virtuallight source, and only this range is assumed to be affected by thevirtual light source.

In step S503, distances R from the central irradiation position 702 oflight from the virtual light source are calculated for respective pixelsof the input image, and are stored in the system memory 122 inassociation with the respective pixel positions. In step S504, for eachpixel position of the input image, weight information RL_map indicatinghow much extent a reflected color component of the virtual light sourceis to be added (reflection characteristic) is calculated by the formula(1), and is stored in the system memory 122.RL_map(p)=α×L·N(p)/D(p)²  (1)

In Formula (1), α is the intensity of the virtual light source, L is adirectional vector of the virtual light source toward the object, N(p)is a normal vector of the object at a pixel position p, and D(p) is adistance between the virtual light source and the object at the pixelposition p.

Hereinafter, a configuration of the relighting processing unit 114 willbe described. FIG. 5 is a block diagram illustrating a configuration ofthe relighting processing unit 114.

In FIG. 5, an RGB signal conversion unit 601 converts the inputluminance/color-difference signals (Y, B−Y, and R−Y) into RGB signals. Ade-gamma processing unit 602 performs de-gamma processing. A virtuallight source adding processing unit 604 adds illumination effects(lighting effects) by the virtual light source to the linear RGB signalssubjected to the de-gamma processing, with reference to thecharacteristics and irradiation range of light from the virtual lightsource, and three-dimensional information of an object image in theirradiation range that are set by the system control unit 50. Adding thelighting effects means adjusting the brightness and the shade/shadow ofthe object image. A correction region attribute detection processingunit 605 detects the attribute of an object region that is to beirradiated with light from the virtual light source.

Here, if an object is, for example, a human, the attribute of the objectregion refers to information indicating the type of the region such ashair, cheeks, nose, pupils, or lips that is classified based on thedetection result of the face/organ detection unit 113. Furthermore, theattributes are classified in association with the reflectioncharacteristics and scattering characteristics when the object isirradiated with light.

A texture characteristic information DB 606 has stored therein multipletypes of texture characteristic information (material appearancecharacteristic information) that correspond to the attributes of theobject regions. A glossy component generation processing unit 607generates a glossy component, which is to be added by irradiating theobject with light from the virtual light source, with reference to theinformation from the correction region attribute detection processingunit 605 and the texture characteristic information DB 606, and theRL_map from the system control unit 50.

A glossy component synthesis processing unit 608 adds, to the image, theglossy component, which is generated on the object image by irradiatingit with light from the virtual light source, with reference to signalsoutput from the virtual light source adding processing unit 604 and theglossy component generation processing unit 607. A gamma processing unit609 performs gamma transformation on signals output from the glossycomponent synthesis processing unit 608. A luminance/color-differencesignal conversion unit 610 converts the RGB signals intoluminance/color-difference signals (Y, B−Y, and R−Y).

Hereinafter, the operation of the relighting processing unit 114 thathas the configuration as above will be described. The relightingprocessing unit 114 reads the luminance/color-difference signals (Y,B−Y, and R−Y) recorded in the image memory 106, and regards them asinputs. The RGB signal conversion unit 601 converts the inputluminance/color-difference signals (Y, B−Y, and R−Y) into RGB signals,and outputs the converted RGB signals to the de-gamma processing unit602. The de-gamma processing unit 602 performs calculation withcharacteristics inverse to the gamma characteristics applied by thegamma processing unit of the image processing unit 105, and converts theRGB signals into linear data. The linear-transformed RGB signals Rt, Gt,and Bt are input to the virtual light source adding processing unit 604.The virtual light source adding processing unit 604 generates correctionsignals for adding relighting effects by the virtual light source to theinput image, and performs correction processing.

Reflected color components (Rv, Gv, and Bv) of the virtual light sourcewhen the image is irradiated with light from the virtual light sourceare calculated using Formulae (2) to (4), with reference to thelinear-transformed RGB signals Rt, Gt, and Bt that are output from thede-gamma processing unit 602, and the RL_map created by the systemcontrol unit 50.Rv(p)=RL_map(p)·f(Rt(p),β)  (2)Gv(p)=RL_map(p)·f(Gt(p),β)  (3)Bv(p)=RL_map(p)·f(Bt(p),β)  (4)

In Formulae (2) to (4), f(Rt(p), β), f(Gt(p), β), and f(Bt(p), β) areobtained by smoothing the reflected color components Rt(p), Gt(p), andBt(p) of the object according to the diffusion property β of the virtuallight source.

The estimated reflected color components (Rv, Gv, and Bv) of the virtuallight source are added to the linear RGB signals that are output fromthe de-gamma processing unit 602 as given in Formulae (5) to (7).Accordingly, output RGB values (RL, GL, and BL) of the processing targetpixels that are irradiated with light from the virtual light source aregenerated.RL(p)=Rt(p)+Rv(p)  (5)GL(p)=Gt(p)+Gv(p)  (6)BL(p)=Bt(p)+Bv(p)  (7)

Here, in the virtual light source adding processing of the virtual lightsource adding processing unit 604, only gain adjustment according to thelight distribution characteristics of the virtual light source isperformed on the reflected color components Rt(p), Gt(p), and Bt(p) ofthe object that are obtained from the image at the time of capturing.Accordingly, if the image signals at the time of image capturing do notinclude a glossy component to be enhanced, it is not possible togenerate a new glossy appearance even by irradiating the image withlight from the virtual light source.

Therefore, the image processing apparatus of the present embodimentperforms, together with relighting, processing for adding and adjustinga glossy appearance of the object surface, with reference tobidirectional reflectance distribution functions. Here, “bidirectionalreflectance distribution function” is defined as the ratio of theintensity of incident light in a lighting direction to the intensity ofreflected light in an observation direction, and refers to informationon which characteristics of the quality of the material of the objectsurface is reflected. Even if object regions are irradiated with thesame light from the virtual light source, the irradiated object regionshave different reflection characteristics on the surface and scatteringcharacteristics in the inside depending on the quality of material ofthe object region, and thus the glossy appearances that are reproducedwill be different. For example, a human object, when being irradiatedwith light, has different glossy appearances between sites such as thehead hair, nose, cheeks, eyes, and lips.

Therefore, in the image processing apparatus of the present embodiment,multiple types of bidirectional reflectance distribution functions arestored in advance in the texture characteristic information DB 606.After the attribute of a partial region of the object is recognized bythe correction region attribute detection processing unit 605, theglossy component generation processing unit 607 selects an optimalbidirectional reflectance distribution function, and generates a glossycomponent that is caused by relighting. Then, the glossy componentsynthesis processing unit 608 performs processing for adding andadjusting an optimal glossy appearance of the partial region of theobject that is subjected to the relighting.

The following will describe the flow of processing (glossy appearancecorrection processing) that is performed on a target pixel by thecorrection region attribute detection processing unit 605 to the glossycomponent synthesis processing unit 608 with reference to the flowchartof FIG. 6.

In step S410, it is determined whether or not a target pixel is withinan irradiation range of light from the virtual light source withreference to the RL_map created by the system control unit 50. If thevalue of the RL_map at the target pixel is greater than or equal to apredetermined value, the target pixel is deemed to be within theirradiation range of light from the virtual light source, and theprocedure advances to step S411. If the value of the RL_map at thetarget pixel is smaller than the predetermined value, the processing forthis target pixel ends.

In step S411, it is determined whether the target pixel belongs to theface, hair, or a specific organ region (nose, eyes, lips, or cheeks) inthe face, with reference to a detection result Fdet of the target pixelobtained by the face/organ detection unit 113, and attribute informationS is generated. It is assumed that, as shown in FIG. 7, values of theFdet and attributes of object partial regions are associated in advancewith each other.

In step S412, the texture characteristic information of the target pixelthat is optimal for the object region is selected from the texturecharacteristic information DB 606 based on the attribute information Sthat is output from the correction region attribute detection processingunit 605. Note that it is assumed that the texture characteristicinformation DB 606 has stored in advance multiple types of texturecharacteristic information that correspond to the face, hair, andspecific organ regions (nose, eyes, lips, and cheeks) in the face. Thetexture characteristic information is defined with, for example, thebidirectional reflectance distribution function.

In step S413, a glossy component for the target pixel is generated basedon the RL_map and the texture characteristic information. In the presentembodiment, fr(p, ω′, ω) is defined as the bidirectional reflectancedistribution function, where p(x, y) is the pixel position, ω′ is thedirection of light of the virtual light source that is incident on thepixel position p, and ω is the direction of light that exits from thepixel position p and is input to the camera. Furthermore, the value ofthe RL_map at the target pixel position is referenced as a level Li(p,ω′) of the light of the virtual light source that is incident on thetarget pixel. Furthermore, a reflection component Lo(p, ω) indicatingthe glossy appearance at the target pixel position is generated by thesystem control unit 50 referencing the normal vector n of the object atthe target pixel position and performing calculation using Formula (7).Lo(p,ω)=∫fr(p,ω′,ω)·Li(p,ω′)·(ω′·n)dω′   (8)

In step S414, the processing for adding and adjusting a glossyappearance of the object surface that is made by relighting is performedby synthesizing the reflected component generated in step S413 with thesignals RL(p), GL(p), and BL(p) output by the virtual light sourceadding processing unit 604.R_out(p)=RL(p)+Lo(p,ω)  (9)G_out(p)=GL(p)+Lo(p,ω)  (10)B_out(p)=BL(p)+Lo(p,ω)  (11)

In the above-described processing, it is possible to analyzecharacteristics of a captured image, and to generate and synthesize aglossy component based on the attribute of a partial region of an objectthat is to be irradiated with light from the virtual light source.Accordingly, it is possible to obtain a more excellent image in whichnot only the brightness and the shade/shadow but also the glossyappearance that is caused when the image is irradiated with light from avirtual light source are adjusted by relighting.

Furthermore, although bidirectional reflectance distribution functionsare used as the information indicating texture characteristics in thepresent embodiment, bidirectional scattering surface reflectancedistribution functions may be used. Furthermore, texture informationthat is generated based on the bidirectional reflectance distributionfunctions and the bidirectional scattering surface reflectancedistribution functions may be stored in advance. Moreover, scatteringcharacteristics within the object and reflection characteristics on anobject surface are different depending also on the wavelengths ofincident light, and thus texture characteristic data may be defined foreach of the R, G, and B signals.

Second Embodiment

Hereinafter, a digital camera according to a second embodiment of thepresent invention will be described with reference to FIGS. 8 and 9. Thedigital camera of the second embodiment differs from the digital cameraof the first embodiment in the operation of a correction regionattribute detection processing unit 705 of the relighting processingunit 114. The following will describe only the difference from the firstembodiment.

Even in the face region, light scattering and reflection characteristicsin the skin or face organs are different between males and females,between adults and children, or between races. Accordingly, in thesecond embodiment, attributes of partial regions of an object areclassified in more detail with reference to not only organ detectionresults but also results of analyzing frequency components and colorinformation of an image. Then, optimal texture characteristicinformation is selected, and glossy components of the object regions areadjusted.

The flow of processing that is performed on a target pixel by thecorrection region attribute detection processing unit 705 to the glossycomponent synthesis processing unit 608 will be described with referenceto the flowchart of FIG. 9.

In step S410, it is determined whether or not a target pixel is withinan irradiation range of light from the virtual light source withreference to the RL_map calculated by the system control unit 50. If thevalue of the RL_map at the target pixel is greater than or equal to apredetermined value, the target pixel is deemed to be within theirradiation range of light from the virtual light source, and theprocedure advances to step S411. If the value of the RL_map at thetarget pixel is smaller than the predetermined value, the processing forthis target pixel ends.

In step S411, a detection result Fdet of the target pixel obtained bythe face/organ detection unit 113 is referenced. In step S812, it isdetermined whether or not the target pixel belongs to a face regionbased on the face/organ detection result Fdet. Since, as shown in FIG.7, the face/organ detection results Fdet and the attributes of theobject regions are associated with each other, if the value of Fdet isother than 0, the procedure advances to step S813, and if the value ofFdet is 0, the procedure advances to step S412.

In step S813, the frequency component and the color information at thetarget pixel are calculated based on the image signals output from thede-gamma processing unit 602. Also, based on the detection result of theface/organ detection unit 113, the frequency component, and the colorinformation, it is determined whether the target pixel belongs to theface, the hair, a specific organ region (nose, eyes, lips, or cheeks) inthe face, and it is furthermore estimated what the race, the gender, andthe age are, and attribute information S is created.

In step S412, the texture characteristic data that is appropriate forthe attribute of the target pixel is selected from the texturecharacteristic information DB 606. For example, if the texturecharacteristic data is selected based on the gender, the texturecharacteristic data that includes more specular reflection componentsthan that for a female is selected for a male. Furthermore, if thetexture characteristic data is selected based on the age, the texturecharacteristic data that includes more diffuse reflection components isselected for a younger object. The operations of steps S413 and S414 arethe same as the first embodiment.

By the above-described processing, the second embodiment also makes itpossible to analyze characteristics of a captured image, and to generateand synthesize a glossy component based on the attribute of a partialregion of an object that is to be irradiated with light from a virtuallight source. Accordingly, it is possible to obtain a more excellentimage in which not only the brightness and the shade/shadow but also theglossy appearance that is caused when the image is irradiated with lightfrom a virtual light source are adjusted by relighting.

Third Embodiment

Hereinafter, a digital camera according to a third embodiment of thepresent invention will be described with reference to FIGS. 10 and 11.The digital camera of the third embodiment differs from the digitalcameras of the first and second embodiments in the operations of aglossy component generation processing unit 807 and a glossy componentsynthesis processing unit 808 of the relighting processing unit 114. Thefollowing will describe only the differences from the first and secondembodiments.

In the third embodiment, a final image quality target is set based on aninstruction from a user, and the glossy component generation processingunit 807 and the glossy component synthesis processing unit 808 arecontrolled so that the glossy appearance that corresponds to theattribute of the object region and the instruction from the user isobtained.

In FIG. 10, the glossy component generation processing unit 807generates a glossy component with reference to, in addition to a signalS output from the correction region attribute detection processing unit705 and the texture characteristic data of the texture characteristicinformation DB 606, a final relighting setting RL_tune that is inputfrom the system control unit 50. The final relighting setting RL_tune isa parameter for adjusting the degree of the glossy appearance that is tobe applied at the time of relighting, and is determined based on aninstruction that was made by the user with the operation unit 120.

The flow of processing that is performed on a target pixel by thecorrection region attribute detection processing unit 705 to the glossycomponent synthesis processing unit 808 will be described with referenceto the flowchart of FIG. 11.

In step S410, it is determined whether or not a target pixel is withinan irradiation range of light from a virtual light source with referenceto the RL_map calculated by the system control unit 50. If the value ofthe RL_map at the target pixel is greater than or equal to apredetermined value, the target pixel is deemed to be within theirradiation range of light from the virtual light source, and theprocedure advances to step S411. If the value of the RL_map at thetarget pixel is smaller than the predetermined value, the processing forthis target pixel ends.

In step S411, a detection result Fdet of the target pixel obtained bythe face/organ detection unit 113 is referenced. In step S812, it isdetermined whether or not the target pixel belongs to a face regionbased on the face/organ detection result Fdet. Since, as shown in FIG.7, the Fdet and the attributes of the object regions are associated witheach other, if the value of Fdet is other than 0, the procedure advancesto step S813, and if the value of Fdet is 0, the procedure advances tostep S412.

In step S813, the frequency component and the color information at thetarget pixel are calculated based on the image signals output from thede-gamma processing unit 602. Also, based on the detection result of theface/organ detection unit 113, the frequency component, and the colorinformation, it is determined whether or not the target pixel belongs tothe face, the hair, a specific organ region (nose, eyes, lips, orcheeks) in the face, and it is furthermore estimated what the race, thegender, and the age are, attribute information S is created, and theprocedure advances to step S1014.

In step S1014, the value of RL_tune, which is a final relighting settingparameter, is transmitted to the glossy component generation processingunit 807 from the system control unit 50 in accordance with aninstruction that was made by the user with the operation unit 120.

In step S1015, the texture characteristic data is selected from thetexture characteristic information DB 606 based on the attributeinformation S of the object region at the target pixel that isdetermined in step S813. For example, if the texture characteristic datais selected based on the gender, the texture characteristic data thatincludes more specular reflection components than that for a female isselected for a male. Furthermore, if the texture characteristic data isselected based on the age, the texture characteristic data that includesmore diffuse reflection components is selected for a younger object.

Furthermore, in step S1015, the characteristic of the diffuse reflectioncomponents of the texture characteristic data that was selected based onthe value of RL_tune instructed by the user is adjusted, and a glossycomponent that is appropriate for the attribute of the object region andthe preference of the user is generated. For example, if the value ofRL_tune is small, the texture characteristic data is adjusted so thatthe value of the diffuse reflection component increases, and a glossycomponent with suppressed gloss is generated. Furthermore, if the valueof RL_tune is large, the texture characteristic data is adjusted so thatthe value of the diffuse reflection component decreases, and a glossycomponent with enhanced gloss is generated. The operations of steps S413and S414 are the same as the first embodiment.

By the above-described processing, the third embodiment makes itpossible to analyze characteristics of a captured image, and to generateand synthesize a glossy component based on the attribute of a partialregion of an object that is to be irradiated with light from a virtuallight source. It is thus possible to adjust the glossy appearanceaccording to the preference of the user. Accordingly, it is possible toobtain a more excellent image in which not only the brightness and theshade/shadow but also the glossy appearance that is caused when theimage is irradiated with light from a virtual light source are adjustedby relighting.

Note that in the first to third embodiments, description was given inwhich a glossy component for a target pixel is generated, but it is alsopossible that a plurality of glossy component images are generated inadvance, and the glossy component image that corresponds to theattribute of a partial region is selected from them to correct theglossy appearance of the partial region.

OTHER EMBODIMENTS

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2015-096920, filed May 11, 2015 which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: atleast one processor and/or circuit configured to function as followingunits: a setting unit configured to set a virtual light source for acaptured image; a brightness correction unit configured to correctbrightness of a partial region of an object in the image using thevirtual light source set by the setting unit; an attribute detectionunit configured to detect an attribute of the partial region; a glossycomponent generation unit configured to generate a glossy component thatis to be applied to the partial region by irradiating the object withlight from the virtual light source, according to the attribute of thepartial region detected by the attribute detection unit and a normalvector of a surface of the partial region; and a glossy appearancecorrection unit configured to correct a glossy appearance of the partialregion using the glossy component generated by the glossy componentgeneration unit.
 2. The image processing apparatus according to claim 1,wherein the glossy appearance correction unit corrects the glossyappearance of the partial region by adding the glossy component to asignal of the image.
 3. The image processing apparatus according toclaim 1, wherein the attribute detection unit detects the attribute ofthe partial region based on light reflection characteristics on asurface of the object, or light scattering characteristics in the insideof the object.
 4. The image processing apparatus according to claim 1,wherein the glossy component generation unit selects, from a pluralityof pieces of material appearance characteristic data, a piece ofmaterial appearance characteristic data that corresponds to the partialregion, and generates the glossy component.
 5. The image processingapparatus according to claim 4, wherein the material appearancecharacteristic data is characteristic data based on a bidirectionalreflectance distribution function.
 6. The image processing apparatusaccording to claim 4, wherein the material appearance characteristicdata is characteristic data based on a bidirectional scattering surfacereflectance distribution function.
 7. The image processing apparatusaccording to claim 1, wherein the glossy component generation unitincludes an instruction unit configured to allow a user to give aninstruction about the glossy appearance.
 8. The image processingapparatus according to claim 1, wherein the attribute detection unitincludes a face detection unit configured to detect a face of an objectfrom the image.
 9. The image processing apparatus according to claim 8,wherein the face detection unit further detects a specific organ of theface of the object.
 10. The image processing apparatus according toclaim 1, further comprising: an estimation unit configured to estimatereflection characteristics of the partial region of the object, whereinthe brightness correction unit corrects the brightness of the partialregion of the object according to the reflection characteristicsestimated by the estimation unit.
 11. The image processing apparatusaccording to claim 1, wherein the glossy component generation unitgenerates the glossy component that is to be applied to the partialregion further according to an irradiation vector of the virtual lightby the setting unit.
 12. The image processing apparatus according toclaim 4, wherein the glossy component generation unit selects, from theplurality of pieces of material appearance characteristic data, thepiece of material appearance characteristic data that corresponds to thepartial region based on the attribute of the partial region detected bythe attribute detection unit.
 13. An image processing method comprising:setting a virtual light source for a captured image; correctingbrightness of a partial region of an object in the image using thevirtual light source set in the setting; detecting an attribute of thepartial region; generating a glossy component that is to be applied tothe partial region by irradiating the object with light from the virtuallight source, according to the attribute of the partial region detectedby the attribute detecting and a normal vector of a surface of thepartial region; and correcting a glossy appearance of the partial regionusing the glossy component generated in the glossy component generating.14. A non-transitory computer readable storage medium storing a programfor causing a computer to execute the steps of the image processingmethod, the image processing method comprising: setting a virtual lightsource for a captured image; correcting brightness of a partial regionof an object in the image using the virtual light source set in thesetting; detecting an attribute of the partial region; generating aglossy component that is to be applied to the partial region byirradiating the object with light from the virtual light source,according to the attribute of the partial region detected by theattribute detecting and a normal vector of a surface of the partialregion; and correcting a glossy appearance of the partial region usingthe glossy component generated in the glossy component generating.